Multimodal Object Detection via Probabilistic Ensembling

Object detection with multimodal inputs can improve many safety-critical systems such as autonomous vehicles (AVs). Motivated by AVs that operate in both day and night, we study object RGB thermal cameras, since the latter provides much stronger signatures under poor illumination. We explore strategies for fusing information from different modalities. Our key contribution is a probabilistic ensembling technique, ProbEn, simple non-learned method fuses together detections multi-modalities. derive ProbEn Bayes’ rule first principles assume conditional independence across Through marginalization, elegantly handles missing modalities when detectors do not fire on same object. Importantly, also notably improves even assumption does hold, e.g., outputs other fusion methods (both off-the-shelf trained in-house). validate two benchmarks containing aligned (KAIST) unaligned (FLIR) images, showing outperforms prior work more than 13% relative performance!